Skip to content

Exchange state halos at every Runge-Kutta stage#377

Open
mwarusz wants to merge 1 commit intoE3SM-Project:developfrom
mwarusz:omega/rk-state-exchng
Open

Exchange state halos at every Runge-Kutta stage#377
mwarusz wants to merge 1 commit intoE3SM-Project:developfrom
mwarusz:omega/rk-state-exchng

Conversation

@mwarusz
Copy link
Copy Markdown
Member

@mwarusz mwarusz commented Mar 25, 2026 •
edited
Loading

#354 added a halo exchange for tracers at every Runge-Kutta stage. To have valid halo data for the del4 velocity tendency, we also need to perform it at every stage for the state variables.

Fixes #376.

Checklist

  • Linting
  • Building
    • CMake build does not produce any new warnings from changes in this PR
  • Testing
    • Add a comment to the PR titled Testing with the following:
      • Which machines CTest unit tests
        have been run on and indicate that are all passing.
      • The Polaris omega_pr test suite
        has passed, using the Polaris e3sm_submodules/Omega baseline
      • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
      • Indicate "All tests passed" or document failing tests
      • Document testing used to verify the changes including any tests that are added/modified/impacted.

@mwarusz
Copy link
Copy Markdown
Member Author

mwarusz commented Mar 25, 2026

Testing

CTest unit tests

  • Machine: aurora
  • Compiler: oneapi-ifx
  • Build type: Release
  • Result: All 39 tests passed
  • Log: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/build-ifx/Testing/Temporary/LastTest.log

  • Machine: frontier
  • Compiler: craygnu-mphipcc
  • Build type: Release
  • Result: All 39 tests passed
  • Log: /lustre/orion/cli115/scratch/mwaruszewski/omega-pr-testing/rk-state-exchng/build-craygnu-mphipcc/Testing/Temporary/LastTest.log

Polaris omega_pr suite

  • Baseline workdir: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/baseline-ifx/
  • Baseline build: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/baseline-ifx/build
  • PR build: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/build-ifx
  • PR workdir: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/pr-ifx
  • Machine: aurora
  • Compiler: oneapi-ifx
  • Build type: <Debug|Release>
  • Log: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/pr-ifx/polaris_omega_pr.o8401396
  • Result:
    • Diffs (1 of 9):
      • ocean/planar/manufactured_solution/convergence_both/del4

Additional tests

On aurora I ran the ocean/planar/manufactured_solution/convergence_both/del4 three times, each with different tasks configuration. The first run was used as a baseline for the second and third. I did this using the polaris Omega submodule and the PR branch. As expected, with the polaris Omega submodule there were baseline failures. On the PR branch there were none. Logs:

polaris_run_1: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/mf-del4-polaris-default/polaris_custom.o8401536
polaris_run_2: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/mf-del4-polaris-goal300/polaris_custom.o8401546
polaris_run_3: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/mf-del4-polaris-goal400/polaris_custom.o8401560

PR_run_1: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/mf-del4-xchng-default/polaris_custom.o8401587
PR_run_2: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/mf-del4-xchng-goal300/polaris_custom.o8401604
PR_run_3: /lus/flare/projects/E3SM_Dec/mwaruszewski/omega-pr-testing/rk-state-exchng/mf-del4-xchng-goal400/polaris_custom.o8401638

@mwarusz mwarusz requested review from brian-oneill and Copilot March 30, 2026 15:22
Copilot AI reviewed Mar 30, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Ensures state-variable halo data is valid at every Runge–Kutta stage so velocity
hyper-diffusion (del4) tendencies are consistent across MPI decompositions,
fixing #376.

Changes:

  • RK4: exchange provisional state halos at every stage (not just a single
    stage).
  • RK2: exchange midpoint state halos before evaluating midpoint tendencies.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
components/omega/src/timeStepping/RungeKutta4Stepper.cpp Exchanges provisional state halos at each RK stage before computing tendencies.
components/omega/src/timeStepping/RungeKutta2Stepper.cpp Exchanges midpoint state halos prior to midpoint tendency evaluation.

@mwarusz mwarusz mentioned this pull request Apr 1, 2026
Merged
7 tasks
brian-oneill
brian-oneill approved these changes Apr 8, 2026
View reviewed changes
Copy link
Copy Markdown

@brian-oneill brian-oneill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, ran the ctests successfully on pm-cpu and frontier CPU & GPU. Approved based on inspection and @mwarusz's polaris testing above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@brian-oneill brian-oneill brian-oneill approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Velocity Del4 tendency is not invariant w.r.t number of MPI tasks

3 participants

@mwarusz @brian-oneill